communityleadershipsummitfandomcom-20200213-history
The OpenStack Continuous Integration Process and How It Shapes the Community
Mark Atwood and Elizabeth Krumbach OpenStack has 500-700 contributors per 30 day window, 200 contributing companies, 170 commits per day every commit is fully tested in a 2.5 hour cycle this process formed in response to earlier anti-patterns eg Java how it works: review.openstack.org check subproj out of git make your change gitreview (CLA must be signed somewhere in here, or you don't get an SSH key) pushes the contribution to a special tree managed by robots, smoke tests are run automatically, results sent up to the metadata Gerrit is a code review system bought by Google and rewritten in Java, works well with Git - event stream: ssh to a port, send JSON commands (OpenStack's version may become a fork as Google is not taking their patches.) anyone can do a +1 or -1 code review for tech nature, comments, git commit message well formatted? "no assholes" rule applies although nitpicking is allowed first level reviewers eventually get social cred and may be recommended by the core reviewers - based on merit, not employer one core reviewer (preferably 2) for each project give a +2, then the change moves into a more extensive test system (2.5 hours) there are no bypasses, process is the same for all "everybody and nobody" has the right to commit tip always works - you can always check out a release and it will work merges happen all the time git with gitreview plug-in Zuul is the gatekeeper with multiple patches, Zuul spins up open compute instances and test trunk + A, trunk + A + B, etc. if all works, all get committed if not, continue combination testing until successful combinations found - those are passed the whole system is another OpenStack project, OpenStack Infra, which can also be changed via the same process how many core reviewers? depends on the project - 12-18 in some OpenStack is a loosely-coupled set of components speaking to each other via REST very decentralized - has to be, due to the membership of the large OpenStack project talking to MediaWiki and KDE about using this system to review this system leads to continuous deployment where do the tests come from? added as part of dev - every feature includes test cases there is a QA team that keeps tests under control so the gate process doesn't become too cumbersome ** this is documented at ci.openstack.org ** use a work in progress tag for intermediate review, non-final code there is a technical committee that decides eg what we're going to test on, therefore what will be in the gate onboarding mechanisms? one recently had a hackathon / trainathon for the CI system other projects do their own onboarding the automation makes minor initial contribs easy so people can onboard themselves there is a practice system that resets every 24 hours "how to get involved" page on the wiki links to the Garrett page you need to know git, and do gitreview rather than git pull to submit it to Garrett newbies don't need to be aware of what's going on behind the scenes if there's a problem, system logs will be sent there are dashboards to see status, most changes are accepted within hours, also sends status emails for each step LaunchPad for bugs, ID, and blueprints much of this is lifted from the Ubuntu process docs maintained as part of source code in git means you can reject a push which does not include docs - docs are done via git if something is not touched in 7 days after a -1, it is marked as "abandoned", but can be resurrected if needed - this is just to keep it out of review queues if nothing is happening on it review day every 2 weeks - see status.openstack.org - to clean up review lists OpenStack uses github as a public mirror - pull requests not accepted directly from github Monty Taylor architected this "We have contributors from the NSA." level 1 reviewers are not necessarily core contributors; level 2 reviewers typically are OpenStack has many contributors who are paid to contribute as their day jobs engagement of corp QA not so great, but they do contribute to Tempest which is a QA project (UTest as a crowd source for testing - check it out) How to set up your own? see the runbook - would take a skilled sysadmin half day of work BUT it does assume that you're running on an OpenStack system, could be difficult if you're not OpenStack installation is not easy at all adoption: data localization is a growing issue with recent US NSA revelations -> local clouds will be popular there is not a public OpenStack cloud in every jurisdiction yet, but there is growing impetus for it #openstackinfra on freenode parallelizable tests preferred, but WIP assuming that computation is infinite and free it does break sometimes, ie because of a new dependency